feat(tooling): add event-monitor arrival-time dashboard - #538
feat(tooling): add event-monitor arrival-time dashboard#538MegaRedHand wants to merge 7 commits into
Conversation
Standalone Rust/axum collector + vanilla-JS dashboard that dials the /lean/v0/events SSE stream of several ethlambda nodes, timestamps each event on one collector clock, and visualizes arrival offset within the slot (rolling per-node beeswarm) and inter-node propagation delay per block/aggregate/head (a second delta beeswarm, delay behind the first node to see each id). Own Cargo workspace with no ethlambda-crate deps; speaks only the documented SSE/HTTP wire shape frozen in CONTRACT.md. Rolling window is adjustable live from the header, a fresh page load backfills recent history via GET /api/history so it is never blank, and ?demo=1 runs the frontend fully offline on synthetic data.
🤖 Kimi Code ReviewOverall: This is a well-architected standalone monitoring tool with clear separation of concerns, good documentation ( 1. Critical: Invalid dependency versionFile: toml = "1.1.3"Issue: The Fix: toml = "0.8"2. Code correctness & robustnessFile: let url = format!(
"{}/lean/v0/events?topics={}",
node.url.trim_end_matches('/'),
topics.join(",")
);Issue: Topics are not URL-encoded. If a topic name contains reserved characters (e.g., Fix: Use let url = format!("{}/lean/v0/events", node.url.trim_end_matches('/'));
let url = reqwest::Url::parse_with_params(&url, &[("topics", topics.join(","))])?;File: axum::serve(listener, app).await?;Issue: No graceful shutdown handling. The server will not respond to SIGINT/SIGTERM cleanly, potentially dropping active SSE connections. Fix: Add signal handling: let server = axum::serve(listener, app);
let shutdown = tokio::signal::ctrl_c();
tokio::select! {
result = server => result?,
_ = shutdown => tracing::info!("shutting down"),
}File: Issue: No validation that Fix: Add validation after loading: if config.nodes.is_empty() {
return Err(anyhow::anyhow!("config must specify at least one node"));
}File: let serve_dir = ServeDir::new(static_dir).fallback(ServeFile::new(index_html));Issue: Fix: Resolve to absolute path: let static_dir = std::fs::canonicalize(static_dir)?;3. Security considerationsFile: Fix: Add use tower_http::cors::CorsLayer;
// ...
Router::new()
.layer(CorsLayer::permissive()) // or restricted to specific origins4. Performance & correctness observationsFile: let canonical = serde_json::to_string(&input).expect("aggregate id input is always valid JSON");Observation: The File: i64::try_from(duration.as_millis()).unwrap_or(i64::MAX)Good: Correctly handles the theoretical overflow case. File: const HISTORY_MAX_EVENTS: usize = 50_000;Good: Hard cap prevents unbounded memory growth during attestation floods. File: rate.textContent = `${status.events_per_sec.toFixed(1)}/s`;Issue: If the backend sends Fix: Add a guard: if (Number.isFinite(status.events_per_sec)) {
rate.textContent = `${status.events_per_sec.toFixed(1)}/s`;
} else {
rate.textContent = "—";
}5. TestingFile: Good: Integration test with a fake SSE server is well-written and covers the normalization pipeline end-to-end. Suggestion: Add a test for the pruning logic in Summary of required changes
The rest are suggestions for hardening. The architecture is sound and the separation between the Rust collector and JS frontend via Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code Review
No consensus-path code is touched here, so fork choice, attestation validation, STF, XMSS, and SSZ correctness are not directly affected by this PR. I couldn’t run Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewReview: PR 538 —
|
Greptile SummaryThis PR adds a standalone dashboard for monitoring event arrival and propagation times. The main changes are:
Confidence Score: 5/5No additional blocking issue qualifies for a new follow-up comment.
|
| Filename | Overview |
|---|---|
| tooling/event-monitor/src/config.rs | Defines monitor settings, node endpoints, and default event topics. |
| tooling/event-monitor/src/hub.rs | Adds live event broadcasting, node status storage, and bounded history backfill. |
| tooling/event-monitor/src/collector.rs | Connects to node event streams and publishes normalized events and status updates. |
| tooling/event-monitor/web/app.js | Bootstraps metadata and history, merges live events, and updates the dashboard. |
Reviews (2): Last reviewed commit: "Merge branch 'main' into feat/event-moni..." | Re-trigger Greptile
| fn default_topics() -> Vec<String> { | ||
| vec![ | ||
| "block".to_string(), | ||
| "attestation".to_string(), | ||
| "aggregate".to_string(), | ||
| ] | ||
| } |
There was a problem hiding this comment.
Default Subscription Is Rejected
The current event endpoint rejects the entire request when any topic is unknown. With these defaults, nodes on main return HTTP 400 for attestation and aggregate, so the collector also loses supported block events and remains reconnecting/down. The default must use currently supported topics or the collector must retry with unsupported topics removed.
Prompt To Fix With AI
This is a comment left during a code review.
Path: tooling/event-monitor/src/config.rs
Line: 57-63
Comment:
**Default Subscription Is Rejected**
The current event endpoint rejects the entire request when any topic is unknown. With these defaults, nodes on `main` return HTTP 400 for `attestation` and `aggregate`, so the collector also loses supported `block` events and remains reconnecting/down. The default must use currently supported topics or the collector must retry with unsupported topics removed.
How can I resolve this? If you propose a fix, please make it concise.Quality cleanups from a /simplify pass; no behavior change and the serialized wire shape is untouched: - serialize HistorySnapshot directly, dropping the field-identical HistoryResponse DTO and its hand copy - add NodeConfig::endpoint() as the single home for the node URL join, used by the collector and the timing bootstrap - fold the checkpoint-topic payload into SlotBlockPayload (serde ignores the extra state field) and trim ReorgPayload to the surfaced fields - extract node_status() for the four repeated status literals - fetch genesis + spec concurrently so a dead node costs one timeout - propagation panel: reject unknown nodes at ingest, drop a dead branch, and cache per-dot jitter + the id's min arrival at insert rather than recomputing both every animation frame
The propagation panel offered a `head` toggle, but head arrival spread is a fork-choice output derived from the same attestation set as `block`, so it told us nothing the block lane didn't already show. Dropping the toggle also lets the collector stop subscribing to a topic no panel renders. `safe_target` and `chain_reorg` were normalized backend-side and then discarded by both panels' topic filters, so they only ever cost a match arm and a payload struct. `chain_reorg` in particular is a point-in-time event rather than a per-node arrival race, so it does not belong on a beeswarm. Keeps CONTRACT.md authoritative: the §2 topic table and the propagation toggle spec now match what the code accepts and renders.
🤖 Codex Code Review
No consensus-layer logic is touched in this PR, so fork choice, attestation validation, STF, XMSS, and SSZ correctness are unaffected here. I couldn’t run Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewI have findings to report, but let me give the full written review directly since the task calls for markdown output, not the structured tool. Review:
|
Reconnect backoff reset was keyed on a clean end-of-stream. The common case is the opposite: a node restart surfaces as a stream error after hours of healthy streaming, so every restart ratcheted the delay one step permanently and after ~7 of them a perfectly healthy node reported `down` and took MAX_BACKOFF to reconnect. Key the reset on how long the session lasted instead, via an `Attempt` that is healthy once it survives one heartbeat. That also closes the inverse hazard the old rule left open: a peer which accepts the request and immediately closes the stream ended "cleanly", hit the reset, and was retried at INITIAL_BACKOFF forever instead of backing off. Both the collector's history ring and the dashboard's rolling window key retention off the highest slot seen, and that watermark only ever moves up. One event from a node on a different genesis therefore aged out every real event and blanked both panels until a restart. Drop events whose slot is more than MAX_FUTURE_SLOTS ahead of the slot the collector's own clock is in, warning once per node per connection so the misconfiguration is visible rather than silent. Only the future side is bounded: old slots are legitimate and common (finalized_checkpoint trails head, a syncing node replays history) and cannot move the watermark. Placing the check in `normalize` covers the history ring and both canvas panels from one spot, since no bogus event reaches the frontend. The frontend applies `status` immediately but buffers `chain` during startup backfill, so the /api/history snapshot could overwrite a fresher live status with an older one. Live status now wins. Test fixtures paired slot-128 and slot-12 payloads with sub-second arrival times, i.e. events far in the future, which the new bound rejects. They now place arrival inside a plausible slot via `arrival_for`, keeping every original assertion; `block_topic_maps_id_to_block_root` asserts CONTRACT §2's worked offset of 123ms rather than a negative offset, which `offset_ms_can_be_negative_under_clock_skew` already covers. CONTRACT.md gains the slot bound in §2 and the live-status-wins rule in §4.
Each tool under tooling/ declares its own [workspace] table, which is what keeps it decoupled from the root workspace but also means every existing check misses it: `cargo fmt --all`, `cargo check --workspace` and `cargo clippy --workspace` all stop at the root workspace members, as do `make lint` and `make test`. tooling/event-monitor therefore landed with nothing verifying it, and would have stayed green only for as long as someone remembered to run cargo by hand in that directory. Add `tooling-lint` (fmt --check + clippy -D warnings) and `tooling-test`, both looping over `tooling/*/Cargo.toml` so a second tool is picked up without touching the Makefile. `set -e` inside the loop is load-bearing: without it a failing clippy would be masked by the exit status of the next command in the loop body. The `[ -e ]` guard keeps the recipes a clean no-op if the glob ever matches nothing. Wire them into `lint` and `test` rather than leaving them standalone, so the documented pre-commit workflow cannot pass locally while CI fails, and extend `fmt` to format tooling too, so `make fmt` does not leave work that `tooling-lint` then rejects. The CI Lint job calls the same two targets instead of restating the commands, and its rust-cache now lists tooling/event-monitor: separate workspaces have their own target dir and Cargo.lock, so without listing it the tooling build is recompiled from scratch every run and its lockfile does not feed the cache key. Tests run in Lint rather than in the Test job because clippy --all-targets has already compiled them there, and because they need none of the Test job's leanSpec fixtures.
Summary
Adds
tooling/event-monitor/, a standalone live arrival-time dashboard forlean-consensus (ethlambda) nodes. It dials the
GET /lean/v0/eventsSSE streamof several nodes, timestamps each event on a single collector clock, and serves
a browser dashboard visualizing:
The rolling window is adjustable live from the header, and a fresh page load
backfills recent history from the collector so it is never blank.
Design
tooling/event-monitor/(empty[workspace]table); it is not a member of the parentethlambdaworkspace and does not affect the main build.CONTRACT.mdis the frozen interface between the Rust/axum backend (src/) and the vanilla-JS frontend (web/, no build step).?demo=1runs the frontend fully offline with synthetic data.This PR contains only the tooling. The RPC / event-bus changes that make a
node emit these events are out of scope here and ship in the chain-events series
(see below).
Required ethlambda API functionality
The monitor consumes chain events over SSE plus two bootstrap endpoints for slot
geometry. Dependency status:
GET /lean/v0/eventsmain?topics=<csv>mainGET /lean/v0/genesis,GET /lean/v0/config/specmainblock,head,justified_checkpoint,finalized_checkpointmainattestation,aggregatesafe_target,chain_reorgblock_gossipRequired for the default dashboard view: #534. Both panels default to the
block/attestation/aggregatetopics.blockandheadalready workagainst
main, butattestationandaggregateneed #534 before the rollingbeeswarm and the aggregate propagation view populate.
Optional enrichment: #533 (
safe_target,chain_reorg) and #535(
block_gossip). The collector parses these topics if a node emits them, butthe dashboard does not require them.
Until #534 lands, the monitor is usable with
topics = ["block", "head"]for ablock-only slot-arrival and propagation view.
Testing
cargo test --releaseintooling/event-monitor/: 29 passed (27 lib + 2 integration), 0 failed.?demo=1offline synthetic mode renders both panels with no live nodes.Run